System and method for correlating camera views

ABSTRACT

A method for correlating views of two or more video camera systems includes obtaining a plurality of data point coordinate sets to represent relative positioning between the camera systems and data point objects in the environment. The views may be correlated through Interpolation or extrapolation using the obtained data point coordinate sets. Devices such as lasers may also be used to correlate views of two or more video camera systems.

PRIORITY

This application claims priority from the disclosure of U.S. ProvisionalPatent Application Ser. No. 60/599,346, entitled “System and Method forImmersive Surveillance,” filed Aug. 6, 2004.

BACKGROUND

The present invention relates in general to video systems, with oneembodiment having an image capture and display system and methodproviding both a wide angle field of view and a narrower field of viewof an environment.

It is often desirable to have the ability to view a large area within anenvironment. It may also be desirable to have the ability to focus in onobjects or events of interest within the environment. Such capabilitiesmay be useful for a variety of purposes, including but not limited tosurveillance for security or military applications. No one prior to theinventors has created or used the invention described in the appendedclaims.

BRIEF DESCRIPTION OF DRAWINGS

While the specification concludes with claims that particularly pointout and distinctly claim the invention, it is believed the presentinvention will be better understood from the following description takenin conjunction with the accompanying drawings, in which like referencenumerals identify the same elements. The drawing and detaileddescription which follow are intended to be merely illustrative and arenot intended to limit the scope of the invention as set forth in theappended claims.

FIG. 1 depicts a diagram of an image capture and display system.

FIG. 2 depicts a perspective view of an image capture system.

FIG. 3 depicts a perspective view of an alternative image capturesystem.

FIG. 4 depicts a perspective view of an alternative image capturesystem.

FIG. 5 depicts a perspective view of an alternative image capturesystem.

FIG. 6 depicts a perspective view of an environment with objectsincluding the image capture system of FIG. 2.

FIG. 7 depicts the environment of FIG. 6 with a plurality of imagecapture systems.

FIGS. 8A through E depict various user interface displays.

DETAILED DESCRIPTION

The following description should not be used to limit the scope of thepresent invention. Other examples, features, aspects, embodiments, andadvantages of the invention will become apparent to those skilled in theart from the following description, which includes by way ofillustration, one of the best modes contemplated for carrying out theinvention. As will be realized, the invention is capable of otherdifferent and obvious aspects, all without departing from the invention.Accordingly, the drawings and descriptions should be regarded asillustrative in nature and not restrictive.

FIG. 1 illustrates an image capture and display system (2) comprising animage capture system (4), a processor (6), and a user interface (8).While a single processor (6) is shown and discussed by way of example,it will be appreciated that any number of processors may be used in anysuitable configuration or arrangement. The image capture system (4)comprises a first and second camera system (10, 20). In the presentexample, the immersive and PTZ camera systems (10, 20) are incommunication with the processor (6), as is the user interface (8). Theimmersive and PTZ camera systems (10, 20) capture images, which areprocessed and/or relayed by the processor (6) to a user via the userinterface (8). In one embodiment, the immersive camera system (10)communicates a first digital video signal to the processor (6), whilethe PTZ camera system (20) communicates a second digital video signal tothe processor (6). The first digital video signal corresponds to a videoimage captured by the immersive camera system (10), while the seconddigital video signal corresponds to a video image captured by the PTZcamera system (20).

The user interface (8) includes a display, which displays at least oneof the images provided by the image capture system (4) to a user. Theuser interface (8) of the present example is further configured toreceive user input, such as instructions or commands from the user, someof which may be transmitted through the processor (6) to one or more ofthe cameras. By way of example, the user input may include commands orinstructions affecting the field of view provided by the PTZ camerasystem (20), such as commands changing the orientation and/or zoom levelof a camera within the PTZ camera system (20). Accordingly, the userinterface (8) comprises a control for orienting the PTZ camera system(20). Other possible user inputs will be apparent to those of ordinaryskill in the art.

As shown, the image capture and display system (2) optionally comprisesa storage device (30) in communication with the processor (6). Examplesof suitable storage devices (30) include hard disk drives, opticaldrives, volatile and non-volatile memory, and the like. The storagedevice (30) may store one or more of the images captured by theimmersive or PTZ camera systems (10, 20). The storage device (30) mayalso store correlation data (32) for correlating the fields of viewprovided by the immersive and PTZ camera systems (10, 20), which may beaccessed by the processor (6) for such correlation. The storage device(30) may additionally or alternatively store a variety of otherinformation, including maps or models of the environment, executablecode for directing the processor (6), information relating to use of thesystem, or any other data. As also shown, the system optionallycomprises a laser (34) and a motion detector (36), each being incommunication with the processor (6).

The camera systems (10, 20), processor (6), user interface (8), andother parts of the image capture and display system (2) may be incommunication via any suitable means, mode, method, or medium. By way ofexample only, the images, inputs such as commands, and/or other data maybe communicated in whole or in part via the Internet, a Local AreaNetwork (LAN), a Wide Area Network (WAN), or any other type of open ordosed network, including combinations thereof. It will also beappreciated that the images, commands, and/or other data may becommunicated in whole or in part via wire or wirelessly, includingcombinations thereof. Still other suitable means, modes, methods, andmedia of communication will be apparent to those of ordinary skill inthe art.

The image capture and display system (2) is operable to capture anddisplay any type of images, including video and still images in a widerange of light spectrums. As used herein, the term “video” generallymeans a sequence of still images from which motion can be discerned.Where such video is frame-based and has a frame rate, for example, theframe rate can vary widely. For instance, in many embodiments, the framerate could be between 30 frames per second and one frame per second.Naturally, these are merely examples and the frame rate could be belowor above this exemplary range. Accordingly, the phrase “video image”shall not be construed as requiring any particular refresh rate orimage-changing frequency. Similarly, the phrase “video camera,”including variations thereof, shall be read to include any type ofdevice operable to capture video images. By way of example only, a videocamera may be a digital video camera comprising a CMOS, CCD, or anyother suitable type of image sensor. Alternatively, a video camera maybe non-digital. Optionally, if a non-digital video camera is used, theimage signal produced by such camera may be converted at some pointduring the process into digital form.

FIG. 2 illustrates an image capture system (4) comprising an immersivecamera system (10) and a PTZ camera system (20). In the present example,the immersive camera system (10) comprises two fisheye lenses (14)positioned back-to-back within a housing (40). In other words, theimmersive camera system (10) of the present example comprises twooppositely facing fisheye lenses (14). In the present embodiment, eachfisheye lens (14) may have at least a hemispherical field of view.Accordingly, the immersive camera system (10) provides a wide anglefield of view of at least a portion of the environment in which theimmersive camera system (10) is situated. Each fisheye lens (14) mayhave its own dedicated camera, or both lenses (14) may share the samecamera. For instance, a video image could be captured with a sharedcamera by frames alternating between each lens (14), or each frame beingshared by images captured from the two lenses (14).

Of course, the immersive camera system (10) may take a variety of forms,including but not limited to the number of cameras and types of lenses.By way of example only, the immersive camera system (10) may comprise asingle camera (12) with a fisheye (14) or other wide angle lens; aplurality of cameras having non-fisheye wide angle lenses; a pluralityof cameras having non-wide angle lenses, such as six cameras having 50mm lenses; or a catadioptric system, such as one of the catadioptricsystems disclosed in U.S. Pat. No. 6,216,619. Still other suitableconfigurations for the immersive camera system (10) will be apparent tothose of ordinary skill in the art.

The phrase “fisheye camera” shall be read to include a camera (12)having a fisheye lens (14). Where the immersive camera system (10)includes back-to-back fisheye lenses (14), such as the immersive camerasystem (10) shown in FIG. 2, it will be appreciated that each of thelenses (14) may provide a field of view that is generally hemispherical.In the present example, the back-to-back hemispherical views provided bythe fisheye lenses (14) provide a generally spherical field of view.

In one embodiment, the immersive camera system (10) provides animmersive view. As used herein, the term “immersive” shall be read todenote any extreme wide angle field of view. For instance, and withoutlimitation, a field of view of 120° or greater in any one dimension isconsidered immersive. It will be appreciated that such a view may beprovided by hemispherical, spherical, torpid, cylindrical, cubic,equirectangular, or other types of images, by way of example only. Thus,an “immersive image” is any still or video image having an immersiveview. Accordingly, while the immersive camera system (10) of the presentexample is operable to provide a spherical field of view, it will beappreciated that other immersive views may also be used, and may beobtained by many types of immersive camera system (10) configurations.

The elongate housing (40) has a rigid circumferential wall (42) definingthe interior and exterior of the housing (40). While the wall isdescribed as “circumferential,” that term should not be read asrequiring the housing to have a circular cross section. By way ofexample only, the housing (40) may have a generally circular, square,rectangular, elliptical, or any other suitable cross section. Two ports(44) are positioned in the housing wall (42) and extend from theinterior to the exterior of the housing (40). Each port (44) isgenerally normal to a centerline running along the interior of thehousing (40). In addition, each port (44) is positioned 180° oppositethe other port (44). A mounting structure is positioned in the interiorof the housing (40) for mounting the fisheye camera(s) within thehousing (40), such that each fisheye lens (14) is aligned with one ofthe ports (44). It will be appreciated that, while an elongate,generally cylindrical housing (40) is shown, a variety of other suitabletypes of housing (40) may be used to house the immersive camera system(10). As shown in FIG. 2, the housing (40) of the present exampleincludes brackets (64) for mounting the image capture system (4) to apost (66). Alternatively, any other suitable housing (40) configurationsor features may be used for mounting the image capture system (4).

The housing (40) includes a substantially transparent convex cover (46)positioned over each port (44), such that the covers (46) cover thefisheye lenses (14). In the present example, the lens covers (46) areconstructed of a plastic material. However, it will be appreciated thatany suitable material may be used. In the present example, the lenscovers (46) have a radius such that they do not act as a lens element aslight passes through to the lens (14). In one embodiment, the lenscovers (46) have a curvature such that the cover is equidistant form thesurface of the corresponding lens (14). It will be appreciated that thelens covers (46) may be configured such that they affect the opticalproperties of the immersive camera system (10). In such an alternativeembodiment, to the extent that such an optical effect is undesirable, itwill be further appreciated that such effects may be addressed throughimage processing. It will also be appreciated that the lens covers (46)may be coated with any material for protective or other purposes. By wayof example only, the lens covers (46) may be coated with a material toprovide a high surface energy for preventing buildup of rainwater andthe like on the lens cover (46). Other suitable housing (40)configurations, including lens cover (46) configurations, will beapparent to those of ordinary skill in the art.

Where an immersive view is provided by the immersive camera system (10),it may be desirable to have the processor (6) perform perspectivecorrection on the image before sending it to the user interface (8). Forinstance, where lines in an immersive image appear to be unnaturallybent or warped, or the image otherwise appears distorted relative tohuman perception, the image may be processed to create a perspectivelycorrected image. The appropriate perspective correction or dewarpingtechnique may vary depending on the type of immersive camera, and manysuch techniques are well known in the art. For instance, in the case offisheye immersive images, such perspective correction may be performedin accordance with the teachings of U.S. Pat. No. 6,185,667, or by othertechniques known in the art. The perspective correction may be performedas to all or part of an immersive image. Thus, as used herein, thephrase “perspectively corrected immersive image,” including variationsthereof, shall be read to include, an immersive image where at least aportion of the immersive image has been perspectively corrected.

The PTZ camera system (20) in the present example comprises apan-tilt-zoom (PTZ) camera (22) system mounted to the housing (40). Inthe present example, the PTZ camera (22) is positioned vertically abovethe immersive camera system (10). However, it will be appreciated thatany suitable relative positioning of the immersive and PTZ camerasystems (10, 20) may be used, and may be proximate or distant from oneanother. The orientation of the PTZ camera (22), such as the pan (e.g.,horizontal orientation) and tilt (e.g., vertical orientation) of the PTZcamera (22) by way of example only, can be controllable remotely by auser. In addition, the level of zoom of the PTZ camera (22) can becontrollable remotely by the user. Accordingly, the PTZ camera system(20) of the present example is controllable by a user with respect to atleast orientation and zoom. It will be appreciated that any aspect ofmovement and/or orientation of the PTZ camera (22) may be provided byany suitable assembly, such as cams, gears, gimbals, motors, pulleys,hydraulics, and the like, by way of example only.

It will be appreciated that the PTZ camera system (20) of the presentexample provides a field of view that is narrower than the field of viewprovided by the immersive camera system (10). Preferably, however, thefield of view provided by the PTZ camera system (20) is of a portion ofthe environment that is within the field of view provided by theimmersive camera system (10). Thus, the immersive camera system (10)provides a wide field of view of the environment, while the PTZ camerasystem (20) provides a narrow field of view of the environment, suchthat the narrow view is within the wide view.

While the figures show the PTZ camera system (20) as comprising a PTZcamera (22), the PTZ camera system (20) may take a variety of differentforms, including but not limited to the number of cameras and types oflenses. By way of example only, the PTZ camera system (20) may comprisea plurality of PTZ cameras (22) or other cameras. Where the PTZ camerasystem (20) comprises a plurality of cameras, each of the cameras neednot be controllable individually with respect to orientation and/orzoom. In such an embodiment, control may be effected, at least in part,by merely switching among views provided by cameras comprising the PTZcamera system (20). In another embodiment, the PTZ camera system (20)comprises a fixed camera aimed at a system of one or more moveablemirrors for changing the region viewed by the camera. Still othersuitable configurations for the PTZ camera system (20) will be apparentto those of ordinary skill in the art.

Those of ordinary skill in the art will appreciate that the PTZ camerasystem (20) may comprise a combination of a digital zoom of the imagecaptured by the immersive camera system (10), in addition to a PTZcamera (22). Such alternate embodiments of the PTZ camera system (20)may include a digital zoom of the image provided by the immersive camerasystem (10) to a point where resolution becomes unsatisfactory, at whichpoint the PTZ camera (22) is oriented and zoomed to approximate thefield of view of the digitally zoomed image, and the processor (6)switches the view sent to the user interface (8) to the view provided bythe PTZ camera (22). Preferably, such switching will be relativelyseamless (e.g., lacking substantial time delay).

It will also be appreciated that the image capture system (4) may take awide variety of forms with respect to the combination of the immersivecamera system (10) and the PTZ camera system (20). Such alternativeforms may include, but are in no way limited to, the embodimentsdepicted in FIGS. 3 through 5. By way of example only, an immersivecamera system (10) having fisheye lenses (14) may be mounted to thesides of a PTZ camera (22), such as the system shown in FIG. 3. In thisembodiment, the center line of sight for the PTZ camera (22) intersectsthe center lines of sight for the two fisheye lenses (14). As the PTZcamera (22) moves, the immersive image may be programmatically shiftedand rotated to match the PTZ camera (22) movement, thereby reducing oreliminating orientation changes in the immersive image. Thus, while theimmersive camera system (10) may be rotating and/or tilting relative tothe environment, the immersive image viewed by the user will appearstationary. One advantage of the camera system example shown in FIG. 3is that parallax between the PTZ camera (22) and immersive camera system(10) is reduced or eliminated.

FIG. 4 shows an image capture system (4) where the PTZ camera (22) isenclosed within a generally dome-like housing (48) providing a slot (50)through which the PTZ camera (22) may view areas of interest within theenvironment. The slot (50) may be occupied or covered by a substantiallytransparent material. It will also be appreciated that the immersivecamera system (10) and PTZ camera system (20) may be held within thesame housing (40).

As shown in the embodiment depicted in FIG. 5, the immersive camerasystem (10) and PTZ camera system (20), may be physically separated.FIG. 5 shows a single camera (12) having a fisheye lens (14) mounted toa ceiling (52), with a PTZ camera (22) mounted nearby to the sameceiling (52). Of course, many other variations of image capture systems(4) may be used, as will be apparent to those of ordinary skill in theart. In addition, it will be appreciated that the image capture system(4) may comprise any number or types of cameras in any suitablepositioning or combinations.

It will also be appreciated that the image capture system (4), in wholeor in part, may be mounted in a variety of locations and to a variety ofplatforms. As used herein, the term “platform” shall be read to includeanything that the image capture system (4) may be mounted to. By way ofexample only, and with reference to FIGS. 6 and 7, the image capturesystem (4) may be mounted to a mobile platform such as a terrestrialvehicle (60), a flying machine (62), or a watercraft (64).Alternatively, the image capture system (4) may be mounted to anon-mobile platform, such as a post (66) or other man-made or naturalstructure, the top or side of a building (68), or to a platform within abuilding (68), such as a wall, ceiling (52), or floor, by way of exampleonly. In addition, other suitable locations and platforms for mountingthe image capture system (4) will be apparent to those of ordinary skillin the art. Of course, the image capture system (4) need not be mountedto a platform at all.

The Image capture and display system (2) may further comprise one ormore lasers (34). It will be appreciated that any suitable type or typesof laser (34) or lasers may be used. By way of example only, one or morelasers (34) may be mounted to, or proximate to, the PTZ camera (22) orother camera. One of the lasers (34) may be oriented such that it issubstantially aligned with or otherwise parallel to the line of sight ofa PTZ camera (22) or other camera. In other words, the laser (34) may beoriented such that it points in the same direction in which the PTZcamera (22) is oriented. In one embodiment, and as shown in FIG. 3, thelaser (34) is located within the same housing (24) as the PTZ camera(22), and the light emitted by the laser (34) passes through an opening(26) positioned proximate to the lens (28) of the PTZ camera (22). Othersuitable orientations of lasers (34) will be apparent to those ofordinary skill in the art.

It will also be appreciated that one or more lasers (34) may be mountedor otherwise located anywhere, including locations separate from the PTZcamera (22) and/or housing (40) where the wide angle camera (12) orcameras are located. Nevertheless, even where a laser (34) is locatedseparate from the PTZ camera (22) and/or housing (40) where the wideangle camera (12) or cameras are located, the laser (34) may beconsidered as being part of the image capture system (4). Other suitablelocations for lasers (34) will be apparent to those of ordinary skill inthe art.

The image capture and display system (2) may further comprise a varietyof other devices. By way of example only, one or more Global PositioningSystem (GPS), Radio Frequency Identification (RFD), or other devices maybe used to determine the positioning of all or part of the image captureand display system (2). In addition, or in the alternative, the imagecapture and display system (2) may include a motion detector (36). Themotion detector (36) may trigger an alarm or other form of notice thatmotion has been detected in the environment. In one embodiment,activation of the motion detector (36) causes the frame rate of thedisplayed video image to speed up. The motion detector (36) may also bein communication with one or more of the cameras of the image capturesystem (4), such that the motion detector (36) triggers the capture ofan image upon detection of motion, or such that the motion detector (36)causes a camera to automatically track an object moving within theenvironment. It will be appreciated that the motion detector (36) may bea conventional motion detector (36) in the form of a device that isphysically separate from the one or more cameras of the system.Alternatively, a motion detector (36) may be effected through theprocessing of images provided by one or more of the cameras by knowntechniques, such as those of successive image/pixel comparison and thelike. Other suitable forms and uses of motion detectors (36) will beapparent to those of ordinary skill in the art. It will also beappreciated that any type of sensor other than a motion detector (36)may be used for similar or other purposes.

The views provided by the immersive camera system (10) and the PTZcamera system (20) may be correlated through a variety of methods. Thus,the views from two or more cameras can be matched. For instance, wherethe views provided by the immersive camera system (10) and PTZ camerasystem (20) have been correlated, a user may choose an object or eventof interest within the view provided by the immersive camera system(10), then command the PTZ camera system (20) to provide a view of thesame object or event of interest. For shorthand purposes, the terms“object” and “event” will be used interchangeably and shall be readinterchangeably and inclusive of plurals. Accordingly, the phrase“object(s) of interest” includes “event(s) of interest” and vice-versa.

Correlation may be performed in a manner that is dependent on thegeography of the environment. Such geography-dependent correlation maybe suitable where, for example, the distance between all or part of theimage capture system (4) and an object in the environment is known andfixed and/or some geometric characteristic of the environment isconstant (e.g., the floor or ground is always flat). However,correlation may also be performed in a manner that is independent of thegeography of the environment. Such geography-independent correlation maybe desired where the image capture system (4) will be mounted to amobile platform or where the environment is dynamic (e.g.,characteristics of the environment are subject to change, the ground orfloor is not flat, the environment is otherwise unknown, etc.).

One challenge in correlating camera views may be accounting forparallax. It will be appreciated that a parallax effect may beencountered by having non-co-linear lines of sight among the pluralityof cameras or camera systems. While the parallax effect, if notaccounted for, may adversely affect the accuracy of correlationattempts, it will be appreciated that the impact of the parallax effecton correlation accuracy may be negligible or otherwise acceptable withrespect to objects and events that are beyond a certain distance fromthe image capture system (4). For instance, where an image capturesystem (4) such as the one shown in FIG. 2 is mounted atop a 15-foothigh post (66), the parallax effect may be negligible with respect toobjects and events that are further than 40 feet away from the post(66). In certain applications it may be desirable to reduce or minimizeparallax by moving the respective lines of sight closer together, whilein other applications parallax may be acceptable or even desirable.

One technique to correlate the views of the immersive and PTZ camerasystems (10, 20) is to essentially ignore the parallax effect. In thisembodiment, the view or image provided by the immersive camera system(10) is mapped, such as by Cartesian, cylindrical, or sphericalcoordinates by way of example only. It will be appreciated that, becausethe map of this example will be of the image or view and not theenvironment, such mapping may be accomplished by knowing directionalcoordinates. When a user selects an object of interest within a viewprovided by the immersive camera system (10), the processor (6)determines or receives the coordinates corresponding to the line ofsight direction. The PTZ camera system (20) may be correlated by usingthe same coordinates. This may be accomplished by “assuming” that theimmersive and PTZ camera systems (10, 20) have the same point of origin,and orienting the PTZ camera system (20) such that its line of sightpasses through the selected coordinates relative to the point of origin“shared” by the immersive and PTZ camera systems (10, 20). This approachmay provide the user with a view from the PTZ camera system (20) that is“close enough” to the selected object of interest, such that the usermay subsequently adjust the orientation of the PTZ camera system (20) asnecessary or desired for a better view of the object of interest by anysuitable means.

Another technique to correlate immersive and PTZ camera systems (10, 20)corrects the effects of parallax. In an embodiment of this technique,data points within the environment are collected and entered in astorage device (30) as correlation data (32). The correlation data (32)is referenced to correct the parallax effects or otherwise account forthe parallax when the user selects an object of interest to be viewed bythe PTZ camera system (20). The collection of data points may beperformed at installation of the image capture system (4), by way ofexample only, and may be desirable where the image capture system (4)will remain in a fixed position relative to the environment during use.Each data point may be taken or collected by noting information relatingto the position of an object (“data point object”)—such as a landmark(70) or building (68) like those shown in FIGS. 6 and 7, by way ofexample only—in the environment relative to each camera or camera systemin the form of coordinate sets (“data point coordinates”) correspondingto each camera or camera system. It will be appreciated that thecoordinates may be only two-dimensional, such that each data pointcoordinate set represents the direction for the corresponding data pointobject relative to the immersive or PTZ camera system (10, 20). Eachdata point coordinate set may also include a coordinate or coordinatesrepresenting the distance of a data point object relative to the camerasystems (10, 20).

By way of example only, each data point object position may be noted bymanually orienting the PTZ camera (22) to view the data point object(e.g., such that the data point object is at the center of the PTZ imageor aligned with crosshairs), then clicking with a mouse on the datapoint object as depicted in the navigable immersive or equirectangularimage (102, 100) captured by the immersive camera system (10). Suchclicking may cause the system to note the data point coordinatesrelative to each camera system (10, 20). This noting may be accomplishedby noting the point in the navigable immersive or equirectangular image(102, 100) on which the user clicked, while also noting the orientationof the PTZ camera (22) (e.g., the orientation of the line of sight ofthe PTZ camera (22)) at the time the user clicked.

However, to make the data point coordinates relative to the PTZ camerasystem (20) more accurate, the user may click on the data point objectas depicted within the PTZ image (104) to note the correspondingcoordinates relative to the PTZ camera system (20). Such clicking mayreplace or update the coordinates (relative to the PTZ camera system(20)) that were noted when the user clicked on the navigable immersiveor equirectangular image (102, 100). This refining or updating may bedesired when the user has difficulty in precisely centering the datapoint object within the PTZ image (104), by way of example only.Alternatively, it may be desired where the user has chosen to click orinadvertently clicked on a particular point within the immersive orequirectangular image (102, 100) that is not precisely at the center ofthe PTZ image (104). Still other possible situations in which it may bedesirable to update or refine data point coordinates relative to the PTZcamera system (20) will be apparent to those of ordinary skill in theart.

In another embodiment, data point coordinate sets are predetermined withrespect to the PTZ camera system (20), such that data point coordinatesets are collected only with respect to the immersive camera system (10)(“zero point calibration”). In this embodiment, the PTZ image (104)includes a crosshairs or other indicator within its center, representingthe line of sight of the PTZ camera (22). With the PTZ camera (22) at aninitial position, the user may visually determine the point in theenvironment over which the crosshairs or other indicator is located,then click on the location with the mouse in one of the images capturedby the immersive camera system (10). In response to the cock, the datapoint coordinates with respect to the immersive camera system (10) willbe noted and associated with the corresponding position or data pointcoordinates with respect to the PTZ camera (22), then the system willposition the PTZ camera (22) to the next predetermined orientation.These steps may be repeated several times until the desired number ofdata point coordinate sets have been collected.

It will be appreciated that, for zero point calibration, the number andpositioning of the predetermined PTZ camera (22) orientations may bepreprogrammed or set by the user. While the word “predetermined” is usedto describe orientations of the PTZ camera (22) during zero pointcalibration, it is meant to include PTZ camera (22) orientations thatare selected at random. It will also be appreciated that the data pointcoordinate sets for the PTZ camera system (20) may correspond with thepredetermined orientations, such that data point coordinate sets neednot be “collected.” In zero point calibration, data point objects maytypically be arbitrary (e.g., whatever the PTZ camera (22) happens to bepointing at when positioned at one of the predetermined orientations).Where a crosshairs or other indicator is used for zero pointcalibration, such an indicator may be displayed only during thisprocess. Suitable variations of zero point calibration will be apparentto those of ordinary skill in the art.

A data point object may be provided by a spot of light or otherdetectable reference provided by a laser (34) that is positioned inclose proximity to the PTZ camera (22) and oriented substantiallyparallel to the line of sight of the PTZ camera (22). In thisembodiment, the PTZ camera (22) is manually or automatically oriented inseveral different orientations at arbitrary or other time intervals. Foreach of these different orientations, the user may find thecorresponding spot of light as shown in the navigable immersive orequirectangular image, and click on the depicted spot of light to notethe coordinates. In one embodiment, the click causes the processor (6)to note the location of the click in the navigable immersive orequirectangular image as the location of the data point object relativeto the immersive camera system (10), while simultaneously noting theorientation of the PTZ camera system (20) as the location of the datapoint object relative to the PTZ camera system (20).

In one embodiment, the immersive camera system (10) includes a filterwithin the optical path of the immersive camera system (10) to assist inthe visual detection or otherwise facilitate detection of the spot oflight provided by the laser (34) on objects in the environment asdepicted in the image provided by the immersive camera system (10). Byway of example only, the filter may be an optical band-pass filter, anarrow band filter, or any other suitable filter. Preferably, the filterwill pass the wavelength of the light provided by the laser (34) suchthat the detectable reference has sufficient contrast to be detectedautomatically. In addition, the filter or filters will preferably bebehind one or more lenses (14) of the immersive camera system (10),although the filter may be positioned within or in front of a lens (14).There will also preferably be a device or method for selectivelyapplying the filter, such as, by way of example only: a mechanicaldevice for placing it in front of or behind a lens, and subsequentlyremoving it; an electronic device for activating and deactivating thefilter; or any other suitable means for selectively applying the filter.

As an alternative to the manual data point coordinate collectiondiscussed above, and particularly where a filter is used, the system mayautomatically collect the data point coordinates by automaticallyorienting the PTZ camera (22) (and, hence, the laser (34)) in differentorientations and automatically noting the corresponding location of thespot of light within the view(s) of the immersive camera system (10).Ways of automatically detecting the spot of light within the view(s) ofthe immersive camera system (10) will be apparent to those of ordinaryskill in the art. Alternatively, a laser (34) and/or filter may be usedto facilitate zero point calibration discussed above, includingvariations thereof.

Thus, in the present example, each data point object will have a set ofdata point coordinates corresponding to its location relative to theimmersive camera system (10), and a set of data point coordinatescorresponding to its location relative to the PTZ camera system (20). Aswith the mapping discussed above, the data point coordinates may be ofany suitable coordinate system, such as Cartesian, spherical, and thelike. However, it may be desirable to collect the coordinatesrepresenting all three dimensions. The data point coordinates may becompiled and stored as correlation data (32) in the storage device (30),by way of example only. After a suitable number of data point coordinatesets have been obtained and stored, the processor (6) may reference thecorrelation data (32) when a user selects an object of interest withinthe view provided by the immersive camera system (10) to be viewed bythe PTZ camera system (20). Through interpolation or extrapolation withthe correlation data (32), the processor (6) may correlate the view ofthe PTZ camera system (20) to the view of the immersive camera system(10), such that the PTZ camera system (20) provides the desired view ofthe selected object of interest. While this type of correlation may besuitably accomplished with but a few data points, it will be appreciatedby those of ordinary skill in the art that the accuracy of thecorrelation may increase with the number of data point coordinate setsobtained. It will also be appreciated that this method of correlationmay be suitable for situations where at least parts of the immersive andPTZ camera systems (10, 20) are not physically located in closeproximity to each other. However, closer proximity of the immersive andPTZ camera systems (10, 20) may lead to less parallax, and reduce thenumber of data point coordinate sets to be collected.

Similarly, it will be appreciated that a laser (34), such as a laserrange finder, or other correlation apparatus may be used to map theenvironment or create a three dimensional model of the environment,which may be represented by data stored in the storage device (30) forreference. This map or model would preferably include the position ofboth the immersive and PTZ camera systems (10, 20). Like the correlationdata (32) described above, the map or model may be referenced by theprocessor (6) in response to user input indicating an object of interestin the depicted environment to be viewed by the PTZ camera system (20).By determining the position of the object of interest on the map or inthe model, the processor (6) may orient the PTZ camera system (20) forviewing the selected object of interest. Still other suitable methods ofobtaining and/or using maps and/or models of the environment, includingthe use of alternative correlation apparatuses, will be apparent tothose of ordinary skill in the art.

Another exemplary method by which the views of the immersive and PTZcamera systems (10, 20) may be correlated includes using objectrecognition algorithms, such as pattern correlation to identify datapoint objects.

Another exemplary method by which the views of the immersive and PTZcamera systems (10, 20) may be correlated includes using a laser (34)and filter to determine the current orientation of the PTZ camera system(20) in order to “manually” orient the PTZ camera system (20) to view anobject of interest (“the laser filter method”). In this embodiment, thelaser (34) is positioned such that it is aligned or is otherwiseparallel with the line of sight provided by the PTZ camera system (20).The immersive camera system (10) includes a filter within the opticalpath of the immersive camera system (10) for easier viewing of thedetectable reference provided by the laser (34) on locations in theenvironment. By way of example only, the filter may be an opticalband-pass filter, a narrow band filter, or any other suitable filter.Preferably, the filter will pass only the wavelength of the lightprovided by the laser (34), or at least a very narrow band ofwavelengths surrounding the wavelength of the light provided by thelaser (34). In addition, the filter or filters will preferably be behindone or more lenses (14) of the camera(s)(12) of the immersive camerasystem (10), although the filter may be positioned within or in front ofa lens (14). There will also preferably be a device or method forselectively applying the filter, such as, by way of example only: amechanical device for placing it in front of or behind a lens, andsubsequently removing it; an electronic device for activating anddeactivating the filter; or any other suitable means for selectivelyapplying the filter.

It will be appreciated that the laser (34) may be operated in anysuitable fashion, such as by being on continuously, by way of exampleonly. Alternatively, where there exits a means for selectivelyactivating or applying the filter, the image capture and display system(2) may be configured such that the laser (34) is activated (e.g.,turned on) substantially contemporaneously with the filter beingactivated, with the laser (34) being deactivated (e.g., turned off)substantially contemporaneously with the filter being deactivated. Stillother suitable relationships between laser (34) operation and filteractivation/application will be apparent to those of ordinary skill inthe art. In addition, it will be appreciated that any suitablecorrelation apparatus or apparatuses may be used as an alternative tothe laser (34) and/or filter.

The laser-filter method, including variations thereof, may be performedby a user first viewing the wide angle image provided by the immersivecamera system (10). Upon detecting an object of interest within the wideangle image, the user may activate the filter. The laser (34) will alsobe activated. With the laser (34) and filter activated, the spot oflight or other detectable reference provided by the laser (34) willpreferably appear clearly on the wide angle image provided by theimmersive camera system (10). Upon seeing the light provided by thelaser (34), which will indicate the line of sight of the PTZ camerasystem (20), the user may manually orient the PTZ camera system (20) tomake the spot of light approach the object of interest, such that theobject comes within the field of view of the PTZ camera system (20).This manual orientation may be done with the laser (34) on and thefilter activated, thereby permitting the user to track the motion of thePTZ camera system (20) by detecting the position and watching the motionof the laser light in one of the wide angle images provided by theimmersive camera system (10). The manual orientation may be performedusing any suitable method or device, such as a joystick, mouse, keypad,or touch screen, by way of example only. When the object of interest iswithin the field of view of the PTZ camera system (20), the filter andlaser (34) may be deactivated. It will be appreciated that the processor(6) may further include a program to provide an indicator within one ofthe wide angle images showing the position of the light provided by thelaser (34) to assist the user in detecting the laser light. By way ofexample only, the program may superimpose an arrow or other indicatorshowing the location of the laser light within the image provided by theimmersive camera system (10).

In another embodiment of the laser-filter method, correlation isautomated, in part, through the use of a real-time feedback loop. Theuser indicates a point or region of interest within the navigableimmersive or equirectangular image (102, 100), such as by clicking witha mouse. Then, the laser (34) and filter are activated, and the systemdetects the position of the spot of light or other detectable referenceprovided by the laser (34) within the image (102, 100) on which the userclicked. The system compares this position of the spot of light to theposition of the point or region of interest indicated by the user. Thesystem may then change the orientation of the PTZ camera (22). After theorientation of the PTZ camera (22) has been changed, or while theorientation is changing, the system may again detect the position of thespot of light, and again compare this position to the position of thepoint or region of interest indicated by the user. This process may berepeated until the system determines that the position of the spot oflight is sufficiently on or within the position of the point or regionof interest indicated by the user. During this process, the filter maybe repeatedly activated and deactivated at a predetermined frequency.This frequency may be synchronized with activation and deactivation ofthe laser (34). While the filter may alternatively remain activatedthroughout the correlation, the repeated activation and deactivation maybe desired for purposes such as maintaining image quality (100, 102), byway of example only.

In one variation, the views of the immersive and PTZ camera systems (10,20) are initially roughly correlated by essentially ignoring theparallax effect, as described above. This initial correlation occurswhen the user indicates a point or region of interest within thenavigable immersive or equirectangular image (102, 100), such as byclicking with a mouse, thereby providing an initial command to orientthe PTZ camera (22). The system then engages in the laser-filter methodusing the feedback loop described above to account for the parallax.

The laser-filter method, including variations thereof, is merely oneexample of how correlation may be performed dynamically and in“real-time.” Other variations of the laser-filter method will beapparent to those of ordinary skill in the art. In addition, by usingthe laser-filter method and its variants, correlation data (e.g., datapoint coordinates) need not be collected. Thus, the laser-filter method,including variations thereof, may be particularly suitable inapplications where the relative view of the environment is subject tochange, such as where at least a part of the image capture system (4) ismounted to a mobile platform.

It will also be appreciated that, where a laser (34) is aligned orotherwise parallel with the line of sight of the PTZ camera system (20),and the distance of separation between the immersive and PTZ camerasystems (10, 20) is known, the location of an object of interest may becomputed using straightforward trigonometry. Such a determined locationmay be used for a variety of purposes, including but not limited toarchival or weapon-targeting purposes, or dispatch of personnel orobjects to the location.

Another exemplary method by which the views of the immersive and PTZcamera systems (10, 20) may be correlated in “real-time” includes usinga correlation apparatus, such as a laser range finder by way of exampleonly, to determine a vector to an object or event of interest in theenvironment, then using trigonometry where the relative positioning ofthe immersive and PTZ camera systems (10, 20) is known or can beobtained. In this exemplary embodiment, relative positioning of theimmersive camera system (10), the PTZ camera system (20), and the laserrange finder is preferably known or determinable. Where a user selectsan object of interest in an image provided by the immersive camerasystem (10), the selection is communicated to the processor (6) via theuser input. Knowing the relative positioning of the immersive camerasystem (10) and the laser range finder, the processor (6) may issue acommand to the laser range finder to determine a vector to the object ofinterest. In response, the laser range finder may determine the vectorand communicate the vector to the processor (6). Knowing the relativepositioning of the laser range finder and the PTZ camera system (20),the processor (6) may then use this vector to issue a command to the PTZcamera system (20) to provide a view of the selected object of interest.It will be appreciated that this method may be particularly suitable forembodiments in which the immersive and PTZ camera systems (10, 20) andthe laser range finder include GPS or similar devices. In thisembodiment, the immersive camera system (10), PTZ camera system (20) andlaser range finder may all be at any distance from each other. It willalso be appreciated that the vector may be communicated to aweapon-targeting system or other system, or to a storage device or otheruser, by way of example only.

In light of at least the foregoing, those of ordinary skill in the artwill appreciate that there exist various methods by which the views ofthe immersive and PTZ camera systems (10, 20) may be correlated.Suitable methods include, but are not limited to, those described above.Suitable methods further include combinations, permutations, andvariations of the methods described above. Alternative correlationdevices, including correlation instruments, apparatuses, andcombinations thereof, will also be apparent to those of ordinary skillin the art. It will also be appreciated that, while correlation has beendiscussed above in the context of correlating views of camera systems(10, 20) within the same image capture system (4), correlation may alsobe performed among two or more image capture systems (4), including butnot limited to correlating the view of the immersive camera system (10)of a first image capture system (4) with the view of a PTZ camera system(20) of a second image capture system (4).

There are various ways in which the images and/or other data obtained bythe image capture system (4) may be presented to a user. Suchpresentation may be through a user interface (8), which may comprise adisplay and a means for receiving one or more user inputs. Severalmerely exemplary display variations are depicted in FIG. 8. As shown,the display may comprise a single viewing device, such as a singlemonitor as shown in FIGS. 8A and 8C through 8E by way of example only;or a plurality of viewing devices, such as the three monitors shown inFIG. 8B by way of example only. Preferably, regardless of the number ofviewing devices, at least two simultaneous images are displayedcomprising at least one view provided by the immersive camera system(10) and at least one view provided by the PTZ camera system (20). Ofcourse, the display may provide a single image at a time. It will alsobe appreciated that, where the display is provided in a windows-basedenvironment, the images may be presented in a single window, or eachimage may be presented in its own window. Alternatively, the images maybe presented in any combinations in more than one window.

The display illustrated in FIG. 8A provides three views to the user, andmay be used where the image capture system (4) comprises an embodimentsimilar to the one shown in FIG. 2, by way of example only. It will beappreciated, however, that a display such as that shown in FIG. 8A maybe used where the image capture system (4) is in any other suitable formor configuration. At the bottom half of the monitor screen, the displayof FIG. 5A displays an equirectangular image (100) from an immersiveview of the environment captured by the immersive camera system (10). Asused herein, the term “equirectangular” shall be read to include anyimage that is generally rectangular, and represents a field of view witha span of approximately 360° in the horizontal, and span that is lessthan or equal to approximately 180° in the vertical. Alternatively,“equirectangular” may include any image that is generally rectangular,and represents a field of view with a span of approximately 360° in thevertical, and span that is less than or equal to approximately 180° inthe horizontal. It is well-known to one with ordinary skill in the arthow to convert spherical immersive images into equirectangular format.

The upper left-hand corner of the display provides a navigable immersiveimage (102). In the present example, this immersive image (102) has aspherical field of view. By the image being “navigable,” as that term isused herein, only a region of the immersive image (102) is displayed ata given time, in accordance with user navigation input. In other words,the user may navigate the immersive image (102) to select the region inthe immersive image (102) to be displayed. For instance and withoutlimitation, such navigation may essentially simulate pan and/or tilt inthe immersive image. Accordingly, a navigable aspect of an immersiveimage (102) provides a user the ability to selectively view regions ofthe immersive image as though the user were controlling a PTZ camera(22), by way of example only. Preferably, the selected portion of theimmersive image (102) may be processed for perspective correction, suchthat the displayed image is a perspectively corrected immersive image.It will be appreciated that a navigable image may be navigated manually(e.g., through the user navigation input), or may be navigated per apreprogrammed sequence (e.g., automatic panning).

The upper right-hand corner of the display provides an image (104) ofthe view obtained by the PTZ camera system (20), such as the PTZ camera(22) in the present example. Accordingly, this image will be referred toherein, for illustrative purposes only, as the PTZ image (104). Thecorresponding user input comprises commands for the PTZ camera system(20), such as PTZ commands. Such commands comprise commands fororienting and/or controlling the level of zoom of the PTZ camera system(20). In other words, the user may orient the PTZ camera system (20),such as the PTZ camera (22), through one of the user inputs. Suchcommands or instructions may be communicated through the processor (6),which may perform a correlation process then issue a command to the PTZcamera (22). The PTZ camera (22) may thereby be oriented in response tothe command to provide a view of the corresponding region of interest inthe form of an oriented PTZ image (104).

In one embodiment, the user input for commanding the PTZ camera system(20) comprises a pointing device, such as a mouse by way of exampleonly. The pointing device may be operable to move an indicator withinone of the wide angle images. The indicator may be an arrow, crosshairs,dot, box, or any other suitable indicator. When the pointing device is amouse, and the indicator is an arrow that is movable on the screen withthe mouse, the user may indicate a region of interest by clicking on themouse when the arrow is positioned within the region of interest asdepicted in one of the wide angle images (100 or 102). Such user inputwill be communicated to the processor (6) for orienting the PTZ camerasystem (20) to capture an image corresponding to the region of interestindicated by the user input.

Thus, the user may orient the PTZ camera (22) by clicking on or near anobject of interest in the equirectangular image (100) or in thenavigable immersive image (102) to indicate a point or region ofinterest. If desired, the user may subsequently re-orient the PTZ camera(22) using the display. Alternatively, the user input may comprise anyother suitable user input device or control for orienting the PTZ camera(22), such as a joystick or microphone for vocal commands by way ofexample only. The user may also zoom in or zoom out with the PTZ camera(22) using any suitable device for accomplishing the same. Suitablevariations of software, hardware, and combinations thereof for effectingPTZ commands will be apparent to those of ordinary skill in the art.

In one embodiment, the user input for orienting and/or zooming the PTZcamera system (20) comprises the use of a mouse to create, move, and/orre-size a box enclosing a rectangular or other region of interest withinthe navigable immersive (102) and/or equirectangular image (100). Asused herein, the phrase “region of interest” includes any region of animage in which an object of interest is located. In this embodiment, theuser may delineate a region of interest in the navigable immersive (102)and/or equirectangular image (100) by moving a pointer with a mouse to acorner of the region of interest within the image. The user may thenpush a button on the mouse, then move the mouse with the buttondepressed until the pointer reaches the opposite corner of the region ofinterest. The box will be created when the user releases the button onthe mouse, and will be defined by the opposite corners thus indicated.In response to the creation of the box (as a user input), the processor(6) may orient the PTZ camera system (20) to provide a PTZ image (104)of the delineated region of interest. In addition to the positioning ofthe box defining or effecting the orientation of the PTZ camera system(20), the size of the box may serve to define or effect the desired zoomlevel of the PTZ camera system (20). The user may delineate anotherregion of interest by following the same steps.

Alternatively, a box or other indicator (108) corresponding to theregion being viewed by the PTZ camera system (20) may always be presenton the navigable immersive (102) and/or equirectangular image (100). Inthis embodiment, the user may orient the PTZ camera system (20) byclicking on the region within the box and “dragging” it with a mouse, asis known in the art. Similarly, the user may control the zoom level ofthe PTZ camera system (20) by re-sizing the box (108), such as byclicking on an edge of the box and “dragging” it with a mouse, as isknown in the art. Still other suitable configurations for effectingcontrol of the PTZ camera system (20) with a box will be apparent tothose of ordinary skill in the art.

Accordingly, the user interface (8) comprises a display and user inputs.The display may comprise any suitable video display or displays. Theuser inputs may comprise immersive navigation and PTZ commands. The userinputs may be part of the display, separate therefrom, or both.

As previously stated, as an alternative to using the equirectangularimage (100) for user input for commanding the PTZ camera (22), or inaddition thereto, the display may permit the user to orient the PTZcamera (22) through the navigable immersive image (102). Preferably, inthis embodiment, the user input for navigating the navigable immersiveimage (102) will differ from the user input for commanding the PTZcamera (22). By way of example only, the PTZ command input may bethrough the mouse, with immersive navigation input being through arrowkeys on a keyboard. Other suitable user input forms and combinationswill be apparent to those of ordinary skill in the art.

As illustrated in FIG. 8B, the three images (100, 102, 104) shown in thedisplay of FIG. 8A may alternatively be presented on three separatescreens or viewing devices. It will be appreciated that the threescreens may be arranged in any suitable order. It will also beappreciated that the images (100, 102, 104) may be provided on anysuitable number of screens, and in any suitable combinations within agiven screen.

As illustrated in FIG. 8C, the display may comprise three immersiveimages (102), in addition to the PTZ image (104). Where the immersivecamera system (10) comprises a pair of back-to-back cameras (12) havingfisheye lenses (14), by way of example only, the three immersive images(102) may be derived from the views obtained by the two cameras (12).Alternatively, the immersive camera system (10) may comprise three pairsof back-to-back cameras (12) having fisheye lenses (14), such that eachimmersive image (102) is derived from the views obtained by one of thepairs. Other suitable configurations will be apparent to those ofordinary skill in the art. Each immersive image (102) may be navigable,non-navigable, or combinations thereof. By way of example only, eachimmersive image (102) may present a different, non-navigable view thatspans approximately 120°. Alternatively, each immersive image (102) maybe navigable dependently or independently with respect to the otherimmersive images (102). Where each immersive image (102) is navigabledependently, the navigation of one immersive image (102) may effect anavigation of the other immersive images (102), such that the systemprevents the same view from being provided in more than one immersiveimage (102) at a given moment. Where each immersive image is navigableindependently, navigation of one immersive image (102) may not affectthe view provided by another immersive image (102) in the display. Othersuitable relationships, features, and configurations of the immersiveimages (102) will be apparent to those of ordinary skill in the art.Preferably, at least one if the immersive images (102) provides a PTZcommand input, such that an object of interest within an immersive image(102) may be indicated by the user for orienting the PTZ camera (22) toprovide a view of the same. In other words, upon detecting an object ofinterest within one of the immersive images (102), the user may click onthe object or otherwise indicate the object. In response, the processor(6) will command the PTZ camera (22) to be oriented to provide a view ofthe object of interest. Of course, any other suitable PTZ command inputmay be used.

The display illustrated in FIG. 8D may be used in an image capture anddisplay system (2) where the PTZ camera system (20) includes a pluralityof PTZ cameras (22), by way of example only. As shown, an immersiveimage (102) is presented in the upper left-hand corner. The immersiveimage (102) may be navigable or non-navigable. The remainder of thedisplay includes a PTZ image (104) from each PTZ camera (22) of thesystem. In this embodiment, PTZ commands may be input through theimmersive image (102), or through any other suitable input device. Whenan object of interest has been indicated, the PTZ images (104) mayprovide views of the object from different angles. Where one or more ofthe PTZ cameras (22) are incapable of viewing the particular object ofinterest for whatever reason (“blind PTZ”), the band PTZ may simply notrespond to the command. Alternatively, the blind PTZ may provide theclosest view of the object of interest as possible. While three PTZimages (104) are shown, it will be appreciated that any number of PTZimages (104) may be displayed. Such number may bear any or no relationto a number of PTZ cameras (22) or other variations of the PTZ camerasystem (20) within the image capture system (4).

The display illustrated in FIG. 8E is similar to the display illustratedin FIG. 8A, with added indicators (106, 108). The equirectangular image(100) includes a navigable immersive indicator (106) and a PTZ indicator(108), while the navigable immersive image (102) includes a PTZindicator (108). The immersive indicator (106) in the equirectangularimage (100) shows the region of the environment being currently viewedin the navigable immersive image (102). It will be appreciated that, asthe immersive image (102) is navigated, the navigable immersiveindicator (106) will move within the eguirectangular image (100) tofollow the navigation. Similarly the PTZ indicator (108) shows theregion being currently viewed by the PTZ camera (22). A PTZ indicator(108) may be shown in both the equirectangular image (100) and thenavigable immersive image (102), or just one of the two images. As withthe navigable immersive indicator (106), the PTZ indicator (108) maymove within the image(s) to follow movement of the PTZ camera (22). Inone embodiment, the PTZ indicator (106) is shown as a box. It will beappreciated that, for purposes of accuracy, the box may not comprisestraight lines and/or right angles. Alternatively, the box may representan approximation of the area being viewed by the PTZ camera (22), andthereby comprise straight lines and right angles. The PTZ indicator(108) may also reflect the level of zoom of the PTZ camera (22). In thebox embodiment, the box may increase in size as the PTZ camera (22)zooms out, and decrease in size as the PTZ camera (22) zooms in. As analternative to a box, a crosshairs or other indicator (108) may beimposed upon the navigable immersive (102) and/or equirectangular image(100). Alternatively, the navigable immersive (102) and/orequirectangular image (100) may be presented in color, with the regioncorresponding to the view of the PTZ camera system (20) being indicatedin black and white (or vice-versa). Of course, any other suitableindicator (108) may be used. Where both a navigable immersive indicator(106) and a PTZ indicator (108) are used, differentiation may beprovided by making the indicators (106, 108) different colors, makingone of sold lines and the other of dotted lines, or by using any othersuitable distinguishing feature(s).

While not shown in the Figures, the display may optionally includeadditional visual representations of any information. By way of exampleonly, the display may include a listing of any or all accessible camerasystems, and provide a user the ability to select any camera system fromthat listing for displaying images capture thereby. It will also beappreciated that the image capture and display system (2) may betime-programmed, such that one or more cameras are set to capture imagesduring certain frames of time. In this embodiment, the display mayprovide a user the ability to program the system for such operation, andmay further display information relating to operation times before,during, and/or after such programming. Notwithstanding time-programming,it will be appreciated that one or more of the displayed images mayinclude a notation relating to the time at which the corresponding imagewas captured, such as current time for current images, or previous timesfor images previously recorded, by way of example only.

It will also be appreciated that the display may display recorded imagesseparately from images currently being captured. Such recorded imagesmay be separate in any suitable way, such as by being in a separateframe, separate window, or on a separate viewing device by way ofexample only.

In another embodiment, the display displays a map of at least a portionof the environment in which at least a portion of the image capturesystem (2) is located. The location of all or part of the image capturesystem (2) may be indicated on the map. Where such a map is used in asystem similar to the one depicted in FIG. 7, where several imagecapture systems (2) are used, the map may provide a user input forselecting image capture systems (2) whose captured images are to bedisplayed, in one embodiment, such user input may be provided by theuser clicking on the location of the desired image capture system (2) asdepicted in the map with a mouse or other device. In another embodiment,each image capture system (2) has a distinct indicator on the map, suchthat the user input may comprise clicking on a corresponding distinctindicator within a listing displayed near the map. Maps may be displayedin any suitable way, such as in a separate frame on the same viewingdevice as one or more of the images, in a separate window on the sameviewing device as one or more of the images, or on a viewing device thatis separate from the viewing device(s) on which the images are beingdisplayed, by way of example only. Other suitable forms, contents, uses,and ways of displaying maps will be apparent to those of ordinary skillin the art.

Where zero point calibration is used, the display may include a displayof a progress bar indicating the user's progress in the zero pointcalibration progress.

It will be appreciated that the display may include image(s) from morethan one image capture system (4). For example, where an arrangementsuch as the one depicted in FIG. 7 is used, the display maysimultaneously include images from two or more of the image capturesystems (4). This may include providing a user the option of selectingamong the plurality of image capture systems (4) for displayingcorresponding captured images. In addition, regardless of the number ofimage capture systems (4) used, a user may be provided the option ofconfiguring the display. This may include permitting the user to selectamong the various configurations illustrated in FIGS. 8A through 8E.Still other display configuration options may be provided to the user.

Having shown and described various embodiments and concepts of theinvention, further adaptations of the methods and systems describedherein can be accomplished by appropriate modifications by one ofordinary skill in the art without departing from the scope of theinvention. Several of such potential alternatives, modifications, andvariations have been mentioned, and others will be apparent to thoseskilled in the art in light of the foregoing teachings. Accordingly, theinvention is intended to embrace all such alternatives, modificationsand variations as may fall within the spirit and scope of the appendedclaims and is understood not to be limited to the details of structureand operation shown and described in the specification and drawings.

1-19. (canceled)
 20. A system comprising: a variety of locations; one ormore cameras being mounted on the locations, each camera for capturingan image; one or more processors being connected to the cameras; and oneor more storage devices being connected to the processors; whereincorrelation data is generated by the processors, using the images outputfrom the cameras, and stored in the storage devices.
 21. The systemaccording to the claim 20, wherein the variety of the locations includea mobile platform, a non-mobile platform, and a building.
 22. The systemaccording to the claim 20, wherein at least one of the cameras generatesPTZ images or narrower images.
 23. The system according to the claim 20,wherein at least one of the cameras generates immersive images or widerimages.
 24. The system according to the claim 1, wherein the cameras arephysically separated.
 25. The system according to the claim 1, wherein asubset of the cameras are located within a housing.
 26. The systemaccording to the claim 1, wherein the cameras receive commands from theprocessors.
 27. The system according to the claim 26, wherein at leastone of the commands is user input.
 28. The system according to the claim26, wherein at least one of the commands orients or controls the levelof zoom.
 29. The system according to the claim 26, wherein at least oneof the commands is vocal information.
 30. The system according to theclaim 20, wherein the correlation data is referenced to account forparallax when an user selects an object of interest to be viewed by thecamera.
 31. The system according to the claim 20, wherein thecorrelation data correlates fields of view provided by the cameras,which may be accessed by the processors for such correlation.
 32. Thesystem according to the claim 20, wherein the correlation data describesa map or model.
 33. The system according to the claim 20, wherein theprocessors correlate a view of one of the cameras to a view of anotherof the cameras.
 34. The system according to the claim 20, furthercomprising one or more motion detectors.
 35. The system according to theclaim 34, wherein the motion detectors detect each frame rate of theimages.
 36. The system according to the claim 20, further comprising oneor more displays for displaying one or more images processed by theprocessors.
 37. The system according to the claim 20, further comprisingone or more interfaces connected to the processors.